Methods and apparatus for customizing user-interface control in existing application

ABSTRACT

Techniques for altering application user-interface controls are provided. More particularly the customization of a new or existing user-interface control in an application user-interface is provided. In one aspect of the invention, a method for customizing user-interface controls of an existing application comprises the recording of a procedure description performed by a user in the application user-interface. A new or modified application user-interface control relating to the procedure description is then installed in the existing application.

FIELD OF THE INVENTION

The invention relates to application user-interface controls and, moreparticularly, to the customization of new or existing user-interfacecontrols in an application user-interface.

BACKGROUND OF THE INVENTION

In order to extend the functionality of existing applications severaltechniques have been proposed that allow an end-user to create a newcontrol for the application user-interface. The user actions associatedwith the new control may be scripted by the user through aspecial-purpose scripting language within the application.

An application that permits control configuration in an existingapplication can be utilized by third-party developers, and can beprovided as a package to be installed at the discretion of an end-user.For example, the GOOGLE (GOOGLE is a trademark of Google Inc.) searchengine uses a toolbar having a set of controls, specified in theJAVASCRIPT language (JAVASCRIPT is a trademark of Sun Microsystems,Inc.), that invokes extension mechanisms within browsers, such asINTERNET EXPLORER (INTERNET EXPLORER is a trademark of Microsoft Corp.)or NETSCAPE (NETSCAPE is a trademark of Netscape Communications). If theuser chooses to install the toolbar into the browser, the JAVASCRIPTlanguage creates a new set of controls within an existing toolbar of thebrowser or hosting application.

There also exist application programs that install controls within otherapplications. An example of such a control is the Watson systemdeveloped at Northwestern University. As part of its installation,Watson adds new controls to several other application programs,including INTERNET EXPLORER and MICROSOFT WORD (MICROSOFT WORD is atrademark of Microsoft Corp.). These controls are configured by theWatson developers and are added only to a small set of known programs.The developers of Watson have explicitly coded the actions that are tobe associated with each control.

While the above mentioned approaches introduce the ability to extend thefunctionality of existing applications, they fail to provide amethodology that allows for the installation of any new or modifiedcontrol into any existing application to suit a specific user. Thus, aneed exists for an improved system for customizing a new or existinguser interface control in an application.

SUMMARY OF THE INVENTION

The present invention provides techniques for altering applicationuser-interface controls and, more particularly, for customizing a new orexisting user-interface control in an application user-interface.

For example, in one aspect of the invention, a method for customizing auser-interface control in an existing application comprises thefollowing steps. First, a procedure performed by a user in theapplication user-interface is recorded. Then, a new or modifiedapplication user-interface control relating to the recorded procedure isinstalled in the existing application.

Advantageously, the inventive technique described in the presentinvention goes beyond the existing systems in at least three ways.First, the inventive technique may allow for the installation of newcontrols into any existing application, not just a single application ora predefined set of applications. Second, the methodology may provide auniform method of specifying the procedure that is bound to a newcontrol. Rather than applying idiosyncratic techniques for eachapplication, the user may simply demonstrate the new procedure exactlyas they would perform it within that application. Third, the inventivetechnique allows not only the addition of new controls to existingapplications, but also the remapping of existing controls, allowingusers to modify their functionality.

These and other objects, features and advantages of the presentinvention will become apparent from the following detailed descriptionof illustrative embodiments thereof, which is to be read in connectionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating procedure demonstration andprocedure binding in an application user-interface, according to anembodiment of the present invention;

FIG. 2 is a flow diagram illustrating a recording procedure methodology,according to an embodiment of the present invention;

FIG. 3 is a flow diagram illustrating a procedure control installationmethodology in an application user-interface, according to an embodimentof the present invention;

FIG. 4 is a flow diagram illustrating an activation and replaymethodology for a procedure bound to a newly-created control, accordingto an embodiment of the present invention; and

FIG. 5 is a diagram illustrating an illustrative hardware implementationof a computing system in accordance with which one or morecomponents/methodologies of the present invention may be implemented,according to an embodiment of the present invention.

DETAILED DESCRIPTIONS OF THE PREFERRED EMBODIMENT

The following description will illustrate the invention using anexemplary data processing system architecture. It should be understood,however, that the invention is instead more generally applicable to anydata processing system in which it is desirable to perform efficient andeffective customization of application interface controls.

As will be illustrated in detail below, the present invention introducestechniques for altering user-interface controls of an application and,more particularly, for customizing a new or existing user-interfacecontrol in an application user-interface. Generally, an applicationrefers to a computer program designed for a specific task or use.

This invention comprises a software component that communicates with theoperating system, capturing and delivering user events, such as keyboardclicks and mouse movements. This component can be thought of as a proxy,sitting between the operating system and the application. Procedures arerecorded by capturing these low-level events as well as informationabout where application controls are located. This enables the recordingcomponent to make inferences about what components within theapplication have been activated.

Once a procedure has been recorded, it may be mapped to a newapplication control. Messages may be sent to the application throughoperating system application programming interfaces (APIs), instructingthe application to insert a new control. The proxy receives notificationwhen the control is activated, executes its recorded procedure, andgenerates the stream of user actions that would have been generated bythe user actually activating these controls. In a similar fashion, theproxy can trap events on existing controls, and execute a proceduremapped to that control.

Referring initially to FIG. 1, a block diagram illustrates proceduredemonstration and procedure binding in an application user-interface,according to an embodiment of the present invention. A user 100interacts with an application 102 through computer input techniques,such as the clicking of a mouse and the pressing of keys. These actionscreate events 120 that are captured by an operating system 104. Withinoperating system 104, events 120 may be translated into higher-levelevents 122, which are then passed to application 102. For example, theevents of pressing down on the button of a mouse, releasing the button,then repeating, can be translated into the higher-level event of adouble click.

A procedure capturer 106 registers 128 with the operating system 104 inorder to receive notification 126 of the events 122 that are passed toapplication 102. Most modern operating systems facilitate the setting of“hooks,” which are callback registrations for particular types ofevents, see, e.g. MICROSOFT WINDOWS (MICROSOFT WINDOWS is a trademark ofMicrosoft Corp.).

In accordance with the invention, the application may also create eventsin addition to the user-initiated events. Application events 124 arepassed to operating system 104 along with requests for system actions.These system actions may include the creation of new windows orcontrols, the deletion of windows or controls, or the changing of thevisual representation of on-screen components. Procedure capturer 106requests 128 to be notified of application events 124 as well. If therequests (hooks) 128 are set prior to the creation of a firstapplication window, the procedure capturer can build a complete model ofall on-screen entities (windows, controls) created by the applicationsimply by capturing all the application-initiated events. Alternatively,requests 128 may be sent to operating system 104 for information abouton-screen entities that have been created by application 102 at somepoint after application 102 has created them. Request 128 may be for allon screen entities, followed by a filtering of those entities notcreated by the application of interest. Hybrid approaches are alsopossible.

Events 126 of a specific procedure captured in the procedure capturerdefine a procedure description 130, which can be stored in a procedurerepository 108.

A control installer component 110 is responsible for retrievingprocedure descriptions 132 from procedure repository 108 and bindingthem to new or modified controls within application 102. This isaccomplished by sending commands 142 to operating system 104,instructing operating system 104 to create and add the new controls ormodify existing controls. An example of the creation of a new control isthe addition of a new button to an existing toolbar within application102. Modern operating systems typically provide APIs that facilitatecreating and adding new controls in this manner. In addition to creatingand registering the new control, control installer 110 will addadditional information modifying procedure description 132 and creatingprocedure description 134 containing details of the new control, such asits location and type. Procedure description 134 is returned toprocedure repository 108.

In accordance with the invention, the new or modified control willtypically have a callback function associated with it. This callback maybe a section of code within a command player 112. Command player 112retrieves procedure description 136 from procedure repository 108. Whenan activation event 140 is received from operating system 104, as aresult of the user having activated the newly installed control, commandplayer 112 executes procedure description 136. As it executes, a seriesof events 138 are generated and sent to operating system 104. Theseevents simulate what the user would have done if she had activated thecontrols that were recorded during the recording phase. In effect,command player 112 is replaying the recorded procedure. The events thatare sent to operating system 104 are automatically passed on toapplication 102, and the desired procedure is carried out by application102.

Referring now to FIG. 2, a flow diagram illustrates a recordingprocedure methodology, according to an embodiment of the presentinvention. This may be considered a detailed description of theinteraction of user 100, operating system 104, application 102, andprocedure capturer 106 in FIG. 1. Once the user has specified that a newcontrol is to be created or modified, the procedure capturer requestsand receives an application user-interface description from theoperating system in step 200. This description contains all of theinformation on application controls that will be required fordetermining which controls have been activated when user events arereceived. This information also includes information on spatial locationand type of control. The description may result from a single request,or from a series of requests. The exact structure of the requests willdepend on the operating system API. For example, there may be a requestfor all root windows followed by requests for children of each rootwindow. The procedure capturer uses the application user-interfacedescription to create a control registry—a data structure that containsthe relevant information about each application control. The controlregistry can be structured in a number of ways, including a hierarchicaltree representation, a linked list, and a table.

After receiving the description at the procedure capturer, the procedurecapturer registers with the operating system in order to receivenotification of certain types of user and system actions in step 202.The user actions are those that are performed during demonstration ofthe procedure, such as mouse clicks and key presses. The system actionsare those that affect the application interface during thedemonstration, such as new control creation and control deletion.

The methodology then enters an iterative loop. In step 204, the userdemonstrates actions of a procedure. A user action may be received atthe procedure capturer in step 206. The procedure capturer thendetermines which control has been activated and the details of thatactivation in step 208. This is accomplished by using information in theuser event or action, such as screen location, in conjunction with thecontrol registry for the specific application. The identity of thecontrol being activated and detail information, such as which item in aselection list has been chosen, is determined. Once the controlactivation information is determined, a generic description of thecontrol activation is mapped in step 210. For example, a specificationbased on identity would be used such as “Press the Add button containedwithin the Admin control panel contained within the top-level windowlabeled ‘user preferences.’” The generic control activation informationis then added to a procedure representation in step 212. The procedurerepresentation contains an entry for each user action, specifying whichcontrol has been activated, and the details of the activation including,for example, items selected and text typed.

If a system event, such as control creation is received in step 214, theProcedure Capturer uses that information to update its control registryin step 216. This is particularly important when an application createsnew controls during execution, for example, pop-up menus. In practice,steps 214 and 216 might run in a separate thread, with appropriatesynchronization mechanisms implemented to prevent access conflicts onthe control registry.

Once the event has been processed, the procedure capturer checks to seeif more user actions are to be recorded in step 218. In general,procedures will be recorded until the user pushes a “stop” button, orotherwise signals the end of procedure recording. If there are more useractions, the methodology returns to step 204. If there are no more useractions, the procedure capturer stores the procedure in step 220. Thisoften occurs in a persistent medium such as a file on hard disk or datastructures in main memory. Finally, a variety of graphical userinterfaces allow the user to select the style and placement of thecontrol to be installed during the recording process. For example, acontrol may be installed as a tool bar button or a menu item.

Referring now to FIG. 3, a flow diagram illustrates a procedure controlinstallation methodology in the application user-interface, according toan embodiment of the present invention. This may be considered adetailed description of the interaction between control installer 110,operating system 104, and application 102 in FIG. 1. A control installerfirst sends a request to the operating system to install the new controlor modify the existing control in step 300. For example, the request maybe to add a new button to an existing toolbar within the application, toadd a new toolbar containing a new button to the top-level applicationwindow, or to add a new item to an existing menu within the applicationinterface.

The installation request contains a callback routine location and/orother activation information that will be required to actually run theprocedure. Using this information, the operating system creates the newcontrol, and installs it in the application user-interface in step 302.Additionally, the operating system registers the callback in step 304,using the location supplied by the control installer.

Referring now to FIG. 4, a flow diagram illustrates an activation andreplay methodology for a procedure bound to a newly-created control,according to an embodiment of the present invention. This may beconsidered a detailed description of the interaction between user 100,operating system 104, application 102, and command player 112 of FIG. 1.When the control for the procedure is activated in step 400 by, forexample, pressing a button or selecting a menu item (installed asdescribed in FIG. 3), the command player receives the callback that hasbeen bound to this control in step 402. The command player thenretrieves the associated procedure description in step 404, requests andreceives the application user-interface structure description from theoperating system, and uses it to build a control registry in step 406.This is identical to step 200 in FIG. 2. The command player registersfor system events in step 408. This is identical to the system eventregistration portion of step 202 in FIG. 2.

The methodology then enters an iterative loop at step 410 where actionsremaining in a procedure are examined. The object on which the action isto be performed (e.g., a button, a menu item, etc.) is mapped in step412 from the description stored in symbolic form in the proceduredescription, into corresponding objects in the application interface.Once the concrete object in the application interface has beendetermined, a message is sent to the operating system to emulateactivation of that control in step 414.

In accordance with the invention, while the actions are being replayed,system events will typically be generated. If a system event, such ascontrol creation, is received in step 416, the command player uses thatinformation to update its control registry in step 418. Steps 416 and418 may run in separate threads, with appropriate synchronizationmechanisms implemented to prevent access conflicts on the controlregistry.

After each action is performed, a check is made to determine whetherthere are more actions in step 420. If there are more user actions inthe procedure, the methodology returns to step 410. If there are no moreuser actions in the procedure, the methodology terminates.

In addition to adding new controls to an existing application, theinventive technique described herein can be used to alter the visibleappearance of existing controls within an application. This kind ofalteration of appearance is commonly known as “skinning.” A set ofalternate controls for existing operations can be created for anyapplication. Underlying application controls may be invoked for thecontrols that are being reskinnned, by simulating the actions (e.g.Mouse click, key press) required to activate that control. Thus, it ispossible to have a control that does not use a prerecorded procedurefrom the procedure repository, but simply passes the events to theoperating system and application. For example, an overlay window mayhave a round button where the original application had a square button.When the round button is pressed by the user, the proxy simulates apress of the original square button.

A variety of techniques can be used to display the “skinned” controls.One possibility is to create an overlay window, which is spatiallyco-registered with the application window. Controls that are skinned canbe rendered and the rest of the overall window can be transparent.Alternatively, the bitmap for the original application window may becopied and used to create a new window with the “skinned” portionsredrawn. Using this technique, new user interfaces can be created thatexpose only a portion of the functionality of the application, or thatadd new controls as described above. Any individual control can have itsfunctionality altered by mapping a new procedure to that control.

Another alternative is to provide an interface that allows the user todesign new controls based on existing controls. The existing controlscould be captured from the screen as bitmaps and then altered usingstandard drawing and painting operations. A final possibility isinstallation of program modules into the proxy that alter the visibleappearance of the controls based on previous user actions or otherevents. For example, control A might be re-rendered in a different colorto indicate unavailability whenever control B has been pressed.

Referring now to FIG. 5, a block diagram illustrates an illustrativehardware implementation of a computing system in accordance with whichone or more components/methodologies of the invention (e.g.,components/methodologies described in the context of FIGS. 1 through 4)may be implemented, according to an embodiment of the present invention.For instance, such a computing system in FIG. 5 may implementapplication 102, operating system 104, procedure capturer 106, controlinstaller 110 and command player 112 of FIG. 1.

It is to be understood that such individual components/methodologies maybe implemented and processed on one such computer system, or on morethan one such computer system. For instance, operating system 104 may beimplemented on one computer system, while procedure capturer 106 may beimplemented on another computer system. In the case of an implementationin a distributed computing system, the individual computer systemsand/or devices may be connected via a suitable network, e.g., theInternet or World Wide Web. However, the system may be realized viaprivate or local networks. The invention is not limited to anyparticular network.

As shown, the computer system may be implemented in accordance with aprocessor 510, a memory 512, I/O devices 514, and a network interface516, coupled via a computer bus 518 or alternate connection arrangement.

It is to be appreciated that the term “processor,” as used herein, isintended to include any processing device, such as, for example, onethat includes a CPU (central processing unit) and/or other processingcircuitry. It is also to be understood that the term “processor” mayrefer to more than one processing device and that various elementsassociated with a processing device may be shared by other processingdevices.

The term “memory,” as used herein, is intended to include memoryassociated with a processor or CPU, such as, for example, RAM, ROM, afixed memory device (e.g., hard drive), a removable memory device (e.g.,diskette), flash memory, etc.

In addition, the phrase “input/output devices” or “I/O devices,” as usedherein, is intended to include, for example, one or more input devices(e.g., keyboard, mouse, etc.) for entering data to the processing unit,and/or one or more output devices (e.g., speaker, display, etc.) forpresenting results associated with the processing unit.

Still further, the phrase “network interface,” as used herein, isintended to include, for example, one or more transceivers to permit thecomputer system to communicate with another computer system via anappropriate communications protocol (e.g., Hypertext TransferProtocol/over Secure Socket Layer (HTTP/S)).

Software components including instructions or code for performing themethodologies described herein may be stored in one or more of theassociated memory devices (e.g., ROM, fixed or removable memory) and,when ready to be utilized, loaded in part or in whole (e.g., into RAM)and executed by a CPU.

Accordingly, as described herein, the present invention providestechniques for altering application user-interface controls and, moreparticularly, for customizing a new or existing user-interface controlin an application user-interface. The inventive techniques of thepresent invention are applicable to a large number of applications suchas any in which new controls are desired to increase the functionalityof the application.

Although illustrative embodiments of the present invention have beendescribed herein with reference to the accompanying drawings, it is tobe understood that the invention is not limited to those preciseembodiments, and that various other changes and modifications may bemade by one skilled in the art without departing from the scope orspirit of the invention.

1. A method for customizing a control of a user-interface of an existingapplication comprising the steps of: recording a procedure descriptioncomprising a series of actions performed by a user in the existingapplication user-interface; and in response to the recording step,customizing the control of the user-interface of the existingapplication by installing a user-interface control specific to theprocedure description in the existing application for automaticexecution of the procedure description and generation of the series ofactions performed by the user when the user-interface control isactivated; wherein the recording and customizing steps are implementedvia one or more software components that are loaded from a memory andexecuted by a processor device; wherein the step of installing thecontrol relating to the procedure description in the existingapplication comprises the steps of: sending a request to an operatingsystem from a control installer to install the control; creating andinstalling the control in the existing application user-interfacethrough the operating system; and registering a callback at theoperating system using a location supplied by the control installer. 2.Apparatus for customizing a control of a user-interface of an existingapplication comprising: a memory; and at least one processor, coupled tothe memory operative to: (i) record a procedure description comprising aseries of actions performed by a user in the existing applicationuser-interface; and (ii) in response to the recording operation,customizing the control of the user-interface of the existingapplication by installing a user-interface control specific to theprocedure description in the existing application for automaticexecution of the procedure description and generation of the series ofactions performed by the user when the user-interface control isactivated; wherein the operation of installing the control relating tothe procedure description in the existing application comprises thesteps of: sending a request to an operating system from a controlinstaller to install the control; creating and installing the control inthe existing application user-interface through the operating system;and registering a callback at the operating system using a locationsupplied by the control installer.
 3. The apparatus of claim 2, whereinthe operation of recording a procedure description performed by a userin the existing application user-interface comprises the steps of:transmitting at least one request for application user-interfacestructure information from a procedure capturer to the operating system;and receiving the application user-interface structure information fromthe operating system at the procedure capturer.
 4. The apparatus ofclaim 2, wherein the operation of recording a procedure descriptionperformed by the user in the existing application user-interfacecomprises the step of registering a procedure capturer with theoperating system to receive notification of user actions and systemactions.
 5. The apparatus of claim 2, wherein the operation of recordinga procedure description performed by the user in the existingapplication user-interface comprises the steps of: receivingnotification of user action within the existing application interface ata procedure capturer; determining a result activated by user action atthe procedure capturer; mapping the activated control into a genericdescription at the procedure capturer; adding the generic description toa procedure representation at the procedure capturer; determining ifthere are more user actions; and storing the procedure representation inthe procedure capturer if there are no more user actions.
 6. Theapparatus of claim 2, wherein the at least one processor is furtheroperative to execute an installed control.
 7. The apparatus of claim 2,wherein the operation of installing the user-interface control furthercomprises the step of altering the appearance of at least one existinguser interface control.
 8. An article of manufacture, stored in one ormore memory devices, for customizing a control of a user-interface of anexisting application comprising computer executable instructions toperform the steps of: recording a procedure description comprising aseries of actions performed by a user in the existing applicationuser-interface; and in response to the recording step, customizing thecontrol of the user-interface of the existing application by installinga user-interface control specific to the procedure description in theexisting application for automatic execution of the proceduredescription and generation of the series of actions performed by theuser when the user-interface control is activated; wherein the step ofinstalling the control relating to the procedure description in theexisting application comprises the steps of: sending a request to anoperating system from a control installer to install the control;creating and installing the control in the existing applicationuser-interface through the operating system; and registering a callbackat the operating system using a location supplied by the controlinstaller.